1. Field of the Invention
The present invention relates to an image processing apparatus and an image processing method and, more particularly, to an image processing apparatus and method for reducing noise in a decoded image without degrading the edge in a picture.
2. Description of the Related Art
In a quantization process performed to compress digital data, the process represented by the following equation is typically performed to input data . EQU c'=int (c/Q+0.5) (1)
where Q is a subrange or step size for quantization and the symbol int () represents a function that performs a truncation process. In a inverse quantization process for restoring the data, the process expressed by equation (2) is performed. EQU C"=C'.times.Q (2)
This process is a linear quantization (linear inverse quantization), and the value of the decoded data n.times.Q (n is an integer) is a middle value over the range of a quantization step as shown in FIG. 10, and represents all input data within the subrange or step defined by the following equation. EQU n.times.Q-Q/2.ltoreq.c&lt;n.times.Q+Q/2 (3)
Non-linear quantization techniques have been proposed in which a smaller quantization step is applied in a higher frequency distribution region as shown in FIG. 12 to reduce a quantizing error in connection with data having a large bias in its distribution as shown in FIG. 11.
FIG. 13 is a block diagram of a conventional moving image coding apparatus. In the moving image coding apparatus 1, a frame memory 12 receives and then stores moving image data to be coded.
A motion vector detector 11 detects a motion vector from the input image stored in the frame memory 12. Available as a motion vector detection method is a block matching in which non-superimposing small segments, each being 16.times.16 pixels (hereinafter referred to as macro block) are subjected to block matching. To achieve a higher accuracy, matching is occasionally performed on a per half-pixel basis.
A motion compensator 20 contains a frame memory 21 and predicts the pixel value at each location in an image to be currently coded based on an image that is coded or locally decoded. The predicted value I'[i,j,t] of a pixel value I[i, j, t] at a location (i,j) in the image that is input at time t is expressed using a motion vector (=(vx(i,j,t)), vy(i,j,t))) corresponding to the location as follows: ##EQU1## where i' and j' are expressed as follows. EQU i'=int (i+vx(i,j,t).times.T) EQU j'=int (i+vy(i,j,t).times.T)
where T is a time difference between the time an image I currently being predicted is input and the time the image in the frame memory is input, and I[i',j',t-T], I[i'+1,j',t-T], I[i',j'+1,t-T], I[i'+1,j'+l,t-T] represent pixel values in the frame memory 21 and int(x) represents the maximum integer not in excess of x.
A differential image generator 13 computes a differential between the value of a pixel to be currently coded and its predicted value by the motion compensator 20, and computes the sum s of the absolute values of the differentials on a per macro block basis as follows: ##EQU2## where MBpq denotes a macro block designated by pq. When the sum given equation (5) is smaller than a predetermined threshold T1, the differential value corresponding to each macro block is output. When the sum given by equation (5) is greater than the threshold T1, the value of the pixel to be currently coded, rather than the differential value, is output. A macro block output by the differential image generator 13 and having a pixel value is referred to as an intra-macro block while macro blocks other than the intra-macro block are referred to as an inter-macro block. The inter-macro block has thus a differential value.
A flag f indicating whether a macro block is an intra-macro block is sent from the differential image generator 13 to a variable length coder/multiplexer 16 and is multiplexed into a bit stream there. The bit stream is then sent to an image adder 19.
A discrete cosine transform (DCT) unit 14 subjects a block of 8.times.8 pixels to a two-dimensional DCT. A quantizer (Q) 15 performs a quantization process to a DCT coefficient c provided by the DCT unit 14 using an appropriate quantization step size Q as expressed by the following equation. EQU c'=int (c/Q) (6)
The quantized DCT coefficient c' is fed to the variable length coder/multiplexer 16 and an inverse quantizer (Q.sup.-1) 17.
The inverse quantizer 17 performs the following inverse quantization process according to a step size equal to the one used by the quantizer 15 . EQU c"=c'.times.Q (7)
An IDCT unit 18 performs inverse DCT process to a pixel block of 8.times.8 pixels of the inverse quantized data.
In response to the flag f supplied by the differential image generator 13, the image adder 19 restores the pixel value from the data output by the IDCT unit 18 and the predicted value provided by the motion compensator 20. When the flag f indicates an intra-macro block, the image adder 19 does nothing particular because the data from the IDCT unit 18 is the pixel value. In contrast, when the flag f indicates an inter-macro block, the image adder 19 adds the predicted value output by the motion compensator 20 to the data from the IDCT unit 18 to restore the pixel value. The restored pixel value is sent to the motion compensator 20 to be stored in the frame memory 21 therewithin.
The variable length coder/multiplexer 16 performs a variable length process to and multiplexes the quantized DCT coefficient output by the quantizer 15, the motion vector v output by the motion vector detector 11 and the flag f output by the differential image generator 13, into a bit stream. The bit stream is then transmitted via a predetermined transmission path or is recorded onto a recording medium.
FIG. 14 is a block diagram of a moving image decoding apparatus which receives and decodes the bit stream from the moving image coding apparatus 1 shown in FIG. 13. In the moving image decoding apparatus 31, a demultiplexer/variable length decoder 41 performs the inverse process of the process performed by the variable length coder/multiplexer 16 in the moving image coding apparatus 1 to restore from the bit stream the quantized DCT coefficient, the motion vector v and the flag f. The resulting DCT coefficient is fed to an inverse quantizer 42, the motion vector v is fed to a motion compensator 45, and the flag f is fed to an image adder 44.
The inverse quantizer 42 and an IDCT unit 43 are equivalent to the inverse quantizer 17 and the IDCT unit 18, respectively, and perform the inverse quantization process expressed by equation (7) and the IDCT process, respectively.
The image adder 44 is equivalent to the image adder 19 in the moving image coding apparatus 1. When the flag f sent from the demultiplexer/variable length decoder 41 indicates an intra-macro block, the image adder 19 simply outputs the data from the IDCT unit 43 as its output. When the flag f indicates an inter-macro block, the image adder 44 adds the predicted value output by the motion compensator 45 to the data from the IDCT unit 43 and outputs the sum as the pixel value of the restored image. These pixel values are stored in a frame memory 46 in the motion compensator 45 to generate a predicted image.
The motion compensator 45 is equivalent to the motion compensator 20 in the moving image coding apparatus 1, and predicts the pixel value of each pixel of an image to be currently decoded, using the motion vector v output by the demultiplexer/variable length decoder 41 and the image stored in the frame memory 46.
FIGS. 15 and 16 are respective block diagrams of a moving image coding apparatus and a moving image decoding apparatus respectively employing a wavelet converter and an inverse wavelet converter. The wavelet converter 51, and inverse wavelet converters 52, 61 replace respectively the DCT unit 14 and IDCT units 18, 43 shown in FIGS. 13 and 14. The construction and operation of each of the motion vector detector 11, quantizer 15, inverse quantizers 17, 42, and motion compensators 20, 45 remain unchanged from those shown in FIGS. 13 and 14. Discussed next is thus the difference between these apparatuses in FIGS. 15 and 16 and the moving image coding apparatus and decoding apparatus performing DCT process respectively shown in FIGS. 13 and 14.
In the same manner as in the apparatus shown in FIG. 13, the differential image generator 13 in the moving image coding apparatus performs the determination of intra-macro block. When it determines that the block is an intra-macro block, the average ave of the pixel values within the macro block is computed as follows: ##EQU3## where N.sub.MBpq represents the number of pixels (16.times.16 pixels) within the macro block.
A differential value that is obtained by subtracting the average ave from each pixel value is fed to the wavelet converter 51. The average ave within the intra-macro block is also sent to the variable length coder/multiplexer 16 and the image adder 19 to be used to restore the image.
When the block is an intra-macro block, the image adder 19 adds the average ave of the macro block sent from the differential image generator 13 to inverse wavelet converted data to restore the pixel value.
When the block is an inter-macro block, the differential image generator 13 outputs the differential between the pixel value and the predicted value from the motion compensator 20 and provides its output to the wavelet converter 51 in the same manner as to the DCT unit. The output is then wavelet converted by the wavelet converter 51. The image adder 19 adds the predicted value of the predicted image generated by the motion compensator 20 to the data converted by the inverse wavelet converter 52 to restore the pixel value.
The wavelet converter 51 wavelet converts one frame of data from the differential image generator 13 and outputs the wavelet converted data on a frame-by-frame basis.
The variable length coder/multiplexer 16 performs a variable length process to and multiplexes the quantized wavelet coefficient quantized by the quantizer 15, the motion vector v output by the motion vector detector 11, the flag f and the macro block average ave output by the differential image generator 13, into a bit stream. The bit stream is then transmitted via a predetermined transmission path or is recorded onto a recording medium.
In the moving image decoding apparatus 31 shown in FIG. 16 as well, the inverse wavelet converter 61, image adder 44 and motion compensator 45 perform the decoding process in the same way as already described.
The demultiplexer/variable length decoder 41 performs the inverse process of the process performed by the variable length coder/multiplexer 16 in the moving image coding apparatus 1 to restore from the bit stream the quantized wavelet coefficient, the motion vector v, the flag f and the macro block average ave. The resulting wavelet coefficient is fed to an inverse quantizer 42, the motion vector v is fed to the motion compensator 45, and the flag f and the macro block average ave are fed to an image adder 44.
The inverse quantizer 42 and inverse wavelet converter 61 perform the inverse processes to the quantized wavelet coefficient in the same way as the inverse quantizer 42 and inverse wavelet converter 61 do in the moving image decoding apparatus 31. To predict each pixel value of an image to be currently decoded, the motion compensator 45 performs the same process as the motion compensator 45 does in the moving image decoding apparatus 31.
In the same way as the image adder 44 works in the moving image decoding apparatus 31, this image adder 44 adds the average ave sent from the demultiplexer/variable length decoder 41 to the inverse wavelet converted data in accordance with the flag f when the block is an intra-macro block.
The differential image generator 13 already outputs the differential value between the input pixel value and the predicted value output by the motion compensator 20 in the moving image decoding apparatus 31; thus, when the block is an inter-macro block, the image adder 44 adds the predicted value of the predicted image already generated by the motion compensator 20 to the inverse wavelet converted data to restore the pixel value.
FIG. 17 is a block diagram of the wavelet converter 51. An image data I[i,j] input by the differential image generator 13 is fed to an analyzing horizontal lowpass filter 71 and an analyzing horizontal highpass filter 101 to be subjected to a frequency band split process in a horizontal direction. Available as wavelet analyzing filters are a linear filter having a coefficient, such as the analyzing horizontal lowpass filter 71 shown in FIG. 18A and the analyzing horizontal highpass filter 101 shown in FIG. 18B.
Analyzing filters for wavelet split shown in FIG. 17 and wavelet synthesizing filters for wavelet synthesizing shown in FIG. 19 are constructed to meet the following equations accurately or approximately. EQU H0(-z)F0(z)+H1(-z)F1(z)=0 (9) EQU H0(z)F0(z)+H1(z)F1(z)=2z.sup.-L (10)
where H0(z), H1(z), F0(z) and F1(z) are respectively a transfer function of analyzing lowpass filters 71, 74, 77, 80, 87, and 104, a transfer function of analyzing highpass filters 75, 82, 84, 89, 101, and 106, a transfer function of synthesizing lowpass filters 123, 129, 143, 152, 158, and 162, and a transfer function of synthesizing highpass filters 127, 147, 149, 156, 166 and 168. The letter L represents an arbitrary integer. These constraint conditions guarantee that a synthesized signal accurately or approximately coincides with the input signal prior to band split. FIGS. 20A and 20B list the coefficients of these synthesizing lowpass filters and highpass filters shown as wavelet synthesizing filters in FIG. 19.
The analyzing horizontal lowpass filter 71 in FIG. 17 extracts from the input image data I[i,j] a horizontal low frequency band signal L[i,j] that is a low-frequency component in the horizontal direction, and outputs it to a horizontal sampling unit 72. The horizontal sampling unit 72 performs a decimation process every sample, and outputs the resulting signal L[i',j] to a memory 73. EQU X[i',j]=X[i,j], i'=i/2 (11)
where X is L.
The memory 73 is a memory circuit that is constructed of a plurality of line memories to hold data both the analyzing vertical lowpass filter 74 and the analyzing vertical highpass filter 75 require, and is provided with line memories of the number equal to the number of filters used for vertical filtering process. For example, for the filters shown in FIG. 18, nine line memories are used for each of the analyzing vertical lowpass filter 74 and the analyzing vertical highpass filter 75, both being of a multi-tap lowpass filter.
To perform frequency band split process in the vertical direction, the analyzing vertical lowpass filter 74 and analyzing vertical highpass filter 75 perform a vertical lowpass filtering process and a vertical highpass filtering process, respectively, to the data L[i',j] stored in the memory 73. The filters used here may be equivalent to the ones for filtering process in the horizontal direction (namely, the analyzing horizontal lowpass filter 71 and analyzing horizontal highpass filter 101).
Signal LL[i',j] generated by the analyzing vertical lowpass filter 74 and signal LH [i',j] generated by the analyzing vertical highpass filter 75 are respectively fed to vertical subsampling units 76, 91, which perform subsampling process to the image in the vertical direction, namely, a decimation process for decimating one line every two lines according to equation (12). EQU X[i',j']=X[i',j], j'=j/2 (12)
where X is LL or LH.
The analyzing horizontal highpass filter 101 separates a horizontal high frequency band signal H[i,j] from the image data I[i,j], and a horizontal subsampling unit 102 performs a decimation process according to equation (11) (X is H), and stores the result in a memory 103.
An analyzing vertical lowpass filter 104 and an analyzing vertical highpass filter 106 perform respectively a lowpass filtering process and a highpass filtering process to the image in the vertical direction in the same way as the analyzing vertical lowpass filter 74 and analyzing vertical highpass filter 75 do.
Signal HL[i',j] generated by the analyzing vertical lowpass filter 104 and signal HH[i',j] generated by the analyzing vertical highpass filter 106 are respectively fed to vertical subsampling units 105, 107, which perform subsampling process to the image in the vertical direction, namely, a decimation process for decimating one line every two lines according to equation (12). The X in equation (12) is HL or HH.
After being subjected to vertical subsampling process through the vertical subsampling units 91, 105, 107, frequency band signals LH[i',j'], HL[i',j'], and HH[i',j'] are output to the quantizer 15 via the wavelet converter 51.
The signal LL[i',j'], which is lowpass filtered in the horizontal direction and vertical direction and output by the vertical subsampling unit 76, is fed to a second-stage analyzing horizontal lowpass filter 77 and an analyzing horizontal highpass filter 84.
The processes, identical to those performed by the analyzing horizontal lowpass filter 71, horizontal subsampling unit 72, memory 73, analyzing vertical lowpass filter 74, analyzing vertical highpass filter 75, and vertical subsampling units 76, 91, are now performed by a analyzing horizontal lowpass filter 77, horizontal subsampling unit 78, memory 79, analyzing vertical lowpass filter 80, analyzing vertical highpass filter 82, and vertical subsampling units 81, 83.
Furthermore, the processes, identical to those performed by the analyzing horizontal lowpass filter 101, horizontal subsampling unit 102, memory 103, analyzing vertical lowpass filter 104, analyzing vertical highpass filter 106, and vertical subsampling units 105, 107, are performed by the analyzing horizontal lowpass filter 84, horizontal subsampling unit 85, memory 86, analyzing vertical lowpass filter 87, analyzing vertical highpass filter 89, and vertical subsampling units 88, 90.
Signals LLLL[i",j"], LHLL[i",j"], HLLL[i",j"], and HHLL[i",j"] thus band split are generated from the signal LL[i',j'].
The output signals of the wavelet converter 51, LLLL[i",j"], LHLL[i",j"], HLLL[i",j"], HHLL[i",j"], and LH[i',j'], HL[i',j'], and HH[i',j'] are wavelet coefficients for respective frequency bands.
In the inverse wavelet converter 52 (identical to the inverse wavelet converter 61) shown in FIG. 19, the frequency band signals provided by the wavelet converter 51 in FIG. 17, namely, the wavelet coefficients LLLL[i",j"], LHLL[i",j"], HLLL[i",j"], HHLL[i",j"], and LH[i',j'], HL[i',j'], and HH[i',j'] are sequentially synthesized to restore the image I[i,j].
More particularly, two-stage band split LLLL[i",j"], LHLL[i",j"], HLLL[i",j"], and HHLL[i",j"] out of the input frequency band signals are first synthesized. These signals are respectively stored in memories 121, 125, 141, 145, each having a plurality of line memories. These memories hold data for the number of lines which are needed by succeeding filters including a synthesizing vertical lowpass filter 123, a synthesizing vertical highpass filter 127, a synthesizing vertical lowpass filter 143, and a synthesizing vertical highpass filter 147.
The number of lines needed is (tap count +1)/2 when the tap count of each vertical filter is an odd number, and is (tap count/2) when the tap count is an even number. For example, when a lowpass filter shown in FIG. 20A is used, two line memories are placed before the synthesizing vertical lowpass filter 123. The number of line memories required by each of the memories 121, 125, 141, 145, 150, 154, 160, and 164 in FIG. 19 is different from the number of line memories required by each of the memories 73, 79, 86 and 103 because a vertical up-sampling process is performed between the memory and the filtering process in FIG. 19.
The vertical up-sampling units 122, 126, 142, and 146 perform a process for inserting one line data having all 0s between any two neighboring lines input according to equation (13). ##EQU4## where X=LLLL, LHLL, HLLL or HHLL.
The frequency band signals LLLL[i",j'], LHLL[i",j'], HLLL[i",j'], HHLL[i",j'] that are provided by the vertical up-sampling units 122, 126, 142, and 146, are respectively interpolated in the vertical direction by the synthesizing vertical lowpass filter 123, synthesizing vertical highpass filter 127, synthesizing vertical lowpass filter 143, and synthesizing vertical highpass filter 147, and are then fed to adders 124, 144.
As already described, the synthesizing filters 123, 127, 143, and 147 along with corresponding analyzing filters 80, 82, 87, and 89 have to satisfy the relationships expressed by equations (9) and (10). If the filter having the coefficients listed in FIG. 18 is used as an analyzing filter, the filter having the coefficients shown in FIG. 20 is used as a synthesizing filter.
Interpolated frequency band signals LLLL[i",j'] and LHLL[i",j'] are summed by the adder 124 and interpolated frequency band signals HLLL[i",j'] and HHLL[i",j'] are summed by the adder 144. The two sums are respectively fed to horizontal up-sampling units 128, 148, where a horizontal up-sampling process is performed according to equation (14). ##EQU5## where X=LLL or HLL.
A synthesizing horizontal lowpass filter 129 and a synthesizing horizontal highpass filter 149 perform respectively a horizontal filtering process to the signals that have been horizontal up-sample processed by the horizontal up-sampling units 128, 148 to interpolate them. The two interpolated signals are summed by an adder 140 to restore a low frequency band signal LL[i',j'].
From a memory 150 to a synthesizing horizontal highpass filter 168, the restored signal LL[i', j'] and the frequency band signals LH[i',j'], HL[i',j'], and HH[i',j'] are subjected to the processes which are identical to those performed by the memory 121 through the synthesizing horizontal highpass filter 149, and the image signal I[i,j] is thus reconstructed.
The picture coding method using the above described DCT and motion compensation technique is adopted in international standards such as MPEG (Moving Picture Experts Group) 1, MPEG2 and H. 263, and is widely used.
When coding is performed using the picture coding method utilizing the DCT and motion compensation technique, the image signal is processed on a block-by-block basis, and a great deal of block noise is generated at a lower bit rate. The coded image, if decoded and reproduced as is, is degraded.
When coding is performed using the picture coding method utilizing the wavelet conversion technique, the image signal is processed on a frame-by-frame basis. Although the generation of block noise is precluded, ringing noise unique to the wavelet coding is generated in a plain area in the vicinity of an edge in a picture. The coded image, if decoded and reproduced as is, is degraded.